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INTRODUCTION 

D istance education permeates the field of professional and con¬ 
tinuing education to such an extent that quality assurance (QA) 
is a topic no distance educator or administrator should avoid. 
Quality assurance is an issue not just for continuing education 
but also for higher education generally. As former UPCE A Executive Direc¬ 
tor Kay Kohl (2010) noted, distance technologies have helped reach work¬ 
ing adults and grow the institutions that serve their continuing education 
needs, and in the process have "disrupted institutional structures in areas 
such as financial aid, quality assessment, and criteria for earning degrees" 
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(p. 15). Given the disruptive impact of distance education and technology 
on understandings of educational quality, this article will provide an over¬ 
view of quality-assurance issues in distance education. In that spirit, we 
will examine topics across four general themes: the external and internal 
dimensions of quality assurance; major initiatives that affected distance 
education quality assurance; the most promising tools and techniques in 
the field; and a look forward to what will likely influence the landscape of 
quality assurance in distance education. 

THE EXTERNAL AND INTERNAL DIMENSIONS OF QA AND THE 
INPUTS-OUTCOMES DEBATE 

The regulatory environment and institutional mission shape how one views 
quality assurance in the context of distance education as does the debate 
about outcomes and inputs as measures of institutional quality. The inter¬ 
play among these is complex and often not aligned. 

External 

Institutions must comply with external accountability mandates and regula¬ 
tory requirements, providing transparent, accessible, and meaningful data 
to various stakeholders that include state and federal agencies. Although 
the former are primarily motivated by a need to protect students as con¬ 
sumers, the current system is a patchwork of antiquated and inconsistent 
regulations created for state-based brick-and-mortar institutions, and can 
hinder institutions offering distance and online learning. One fortunate 
aspect of the recent federal regulation confirming state authorization for 
distance education is that efforts are underway to develop a common set 
of standards or a common application process as, for example, the Multi- 
State Reciprocity in Postsecondary Approval and Regulation Project. This 
project is being led by the Presidents' Forum and is funded by the Lumina 
Foundation. In spite of these efforts, however, federal regulation may still 
generate obstacles to innovation, as the regulations focus on instrumental 
measures such as seat time. Current Title IV requirements that mandate 
regular and substantive interaction between faculty and students could 
stand in the way of learning technologies in which learners access content 
directly without faculty as mediator or facilitator. 

There are also less formal external forces in play as, for example. 
Transparency by Design (TbD), a voluntary cooperative in which member 
institutions provide program-level outcomes data, student demographics. 
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program information, as well as results of nationally normed assessments 
such as the Priorities Survey of Online Learning and the National Survey of 
Student Engagement. WCET / WICHE TbD manages data from institutions 
and report them on a public website. College Choices for Adults (www. 
collegechoicesf oradults .org). 

Internal 

A second frame—and to our minds a much more meaningful one—from 
which to view quality assurance is that of internal continuous improvement. 
The ultimate measure of quality is the degree to which students can demon¬ 
strate learning outcomes at the level deemed appropriate for the course and 
the degree regardless of delivery mode. Assessment of distance-education 
course and program outcomes on multiple levels through valid instruments 
and methods is the kind of quality assurance effort that should receive most 
of our attention as educators. Technology-enabled courses (or even hybrid 
courses) have an advantage over conventional face-to-face courses because 
of the tools that can be incorporated for students to demonstrate learning 
and the data that can be collected not only about individual student learn¬ 
ing but also for subsequent course and program improvement. Fortunately, 
in terms of outcomes assessment as the ultimate indicator of institutional 
effectiveness and quality, both regional- and programmatic-accrediting 
bodies are driving all programs—including distance education—in the 
right direction. 

Inputs 

The traditional quality indicators most emphasized by the external envi¬ 
ronment have been input-driven. The debate on inputs versus outputs as 
measures of institutional quality is still prevalent in higher education. The 
long-dominant input approach is steeped in a view of institutional and 
programmatic quality that assumes that putting appropriate people and 
resources into an institution—qualified faculty, qualified and motivated 
students, solid academic resources, student services, and sufficient fund¬ 
ing—leads to the desired outcomes in the form of educated citizens, research 
to improve lives, and service to communities. Ranking systems encourage 
institutions to focus on inputs in order to make them more attractive to 
students, passing off institutional prestige as a measure of quality. Our 
experience in higher education has demonstrated the fallacies of these 
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assumptions as questions arise about how well our graduates have been 
educated and the magnitude of the return from the vast amounts of state, 
national, and private investment. 

Outcomes 

An encouraging development is that both external and internal stakeholders 
are increasingly demanding outcomes data to assess institutional quality. 
The US Department of Education got headlines when it cast a broad net 
to capture the impact of distance-education providers, especially the for- 
profit sector. But less publicized are the efforts of institutions to examine 
student trajectories throughout their college years by such means as the 
National Survey of Student Engagement (NSSE). More than 1,300 colleges 
and universities utilized this self-assessment of student learning and stu¬ 
dents' personal development, which is another measurable component of 
educational quality. It has since announced an update to its survey because 
"higher education is constantly changing, with increasing demands for 
diagnostic and actionable data, and rapid adoption of new technologies 
for ... distance learning programs" (NSSE, 2011). The Collegiate Learning 
Assessment (CLA) was developed to assess "core outcomes espoused by 
all of higher education—critical thinking, analytical reasoning, problem 
solving and writing" (Hersch, 2007, p. 6). Among the most comprehensive 
efforts to measure the actual learning of our students, the CLA "promotes a 
culture of evidence-based assessment in higher education" (Grigsby, 2009, 
p. 58-59). With the release of Academically Adrift (Arum & Roksa, 2011), the 
Council for Aid to Education applauded the message that adherence to 
high expectations of quality and academic rigor does matter, but that the 
publication's "overall portrait of the quality of undergraduate education 
is deeply disturbing" (Benjamin, 2011). 

We are encouraged that external stakeholders are increasingly demand¬ 
ing outcomes data as much as inputs. So one would hope that the two 
perspectives and needs for quality assurance can merge in frameworks that 
serve both external accountability requirements as well as internal needs 
for continuous improvement. Indeed, some scholars have suggested that 
rather than focusing on how these external and internal assurance measures 
are different and sometimes conflicting, administrators should think about 
combining internal and external perspectives to create a lasting and holistic 
culture of quality for an entire institution (Ehlers, 2009). For example, ex- 
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ternal accountability requirements can provide "a reason to explore ways 
to improve the institution...[and] an opportunity to design an inquiry to 
address institutional needs (Behr & Walker, 2009, p. 2). 

SIGNIFICANT QA INITIATIVES 

A variety of quality assurance initiatives promise to influence the design of 
distance teaching and learning, including best-practice frameworks, quality 
course rubrics and checklists, and regular discussion forums on teaching. 

Best-practice frameworks 

The WICHE Cooperative for Educational Telecommunications (WCET), 
University of Texas TeleCampus, and the Instructional Technology Council 
have put together "Best Practice Strategies to Promote Academic Integrity 
in Online Education," one of the most notable lists of best practices (2009). 
The Sloan Consortium (http: / / sloanconsortium.org/quality_scoreboard_ 
online_program) has recently published the "Quality Scorecard for the 
Administration of Online Programs," which contains 70 quality indica¬ 
tors that can aid in the design and evaluation of online programs as well 
as help demonstrate programmatic quality to university administration, 
governance boards, and accrediting bodies. Best practices in online learning 
have matured from segregated "instructional technology" teaching methods 
or standards developed from vendors such as BlackBoard's Greenhouse 
Awards Program, to integrated best practices based on sound pedagogical 
principles that transcend modes of delivery. 

Quality course rubrics and checklists 

The University of Northern Colorado (UNC) has adopted a program that 
reaches deeper into the practices of faculty by diagnosing rather than 
evaluating. In the online nurse-education programs of UNC's College of 
Nursing PhD and Doctor of Nursing Practice degree programs, faculty 
have embraced peer review. The institution offers grant support to its 
faculty for presentations and publications (Dougherty and Roehrs, 2009). 
Current research among the nursing faculty and the CETL focuses on the 
effectiveness in peer review of the QM-facilitated degree-planning process. 
The faculty themselves are leading the effort. At UNC's Monfort College of 
Business, the dean and faculty support a culture of assessing quality from 
within using self-checks in standards, resulting in the Malcolm Baldridge 
National Quality Award. 
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Discussion forums 

The University of Colorado (CU) at Boulder has initiated the Collaborative 
Preparing Future Faculty Network (COPFFN) through its long-established 
Graduate Teacher Program. It serves as a check for quality among gradu¬ 
ate teaching assistants and prepares doctoral students for the professori¬ 
ate through professional development in teaching and research. The 14th 
Annual COPFFN Forum featured multiple sessions on "Assessment Issues 
for Teachers in Colleges and Universities" and featured directors, adminis¬ 
trators, faculty, and other specialists from CU and neighboring institutions 
(COPFFN, 2011). Why did the COPPFN deem assessment at the classroom, 
program, and institutional levels a key issue for the forum? The answer may 
be that just as doctoral-granting institutions (particularly teacher colleges) 
are obligated to prepare future faculty with new strategies for implementing 
quality in a web-based teaching environment, it would be a disservice not 
to inform them of the institutional culture and climate of assessment and 
accountability awaiting them as full-time professionals in higher education. 

NOTABLE QA TOOLS 

Learning analytics are methodologies for capturing data that are aimed at 
improving the conditions for teaching and learning. The methodologies 
include using a variety of technological tools, analytical models, and add¬ 
on features to course-management systems. Learning analytics have been 
most often used to identify students at risk so that pedagogical or other 
interventions can be applied (e.g., the MAP Works-Making Achievement 
Possible Program). Flowever, the promise of learning analytics could well 
involve the identification of student learning styles and abilities that can lead 
to customizing teaching methods, materials, and curricula for individual 
students or small groups of students (Johnson, Smith, Willis, Levine, and 
Haywood, 2011). 

Perhaps the leading factor driving the development and popularity 
of learning analytics is that the e-learning environment is providing an 
unprecedented opportunity to collect and analyze data to support quality 
improvement efforts in student learning and retention. Higher education 
is increasingly applying data mining and analytics long used by industry. 
Institutions can develop predictive models by analyzing massive data sets 
to predict if a student will be successful in an online course and to target 
intervention strategies. They are often aided in these efforts by software 
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solutions developed by private industry or they are developing proprietary 
applications. Data analytics can be used to support quality assurance by 
providing data-driven information for improvements at the course, pro¬ 
gram, and student-support levels. And the best applications of data mining 
can provide actionable reports to faculty, students, and staff to develop and 
implement targeted strategies and improvements and to assess the effec¬ 
tiveness and quality of interventions and improvements. Data mining from 
most learning management systems (LMSs) can show where students spend 
their time and how this behavior relates to student success, as measured as 
course completion and learning-outcomes attainment. Data analytics enable 
institutions to drive quality improvements through valid data, not merely 
on the basis of anecdotal information or intuition. Examples of data mining 
and data analytics include projects from Starfish Retention Solutions (see 
www.starfishsolutions.com), Purdue University's "Signals" program (see 
Arnold, 2010), the Iowa Community College Online Consortium (ICCOC), 
and the Bill and Melinda Gates Foundation. 

The ICCOC project utilizes Pearson/eCollege's reporting software 
called "Enterprise Reporting", which allows customized reports on various 
types of student activity and the continuous collection of data related to that 
activity. ICCOC leaders used the learning-analytics reporting in conjunction 
with a program of student and academic support interventions, and dur¬ 
ing the fall 2005 to fall 2009 period course completion rates went from 77 
to 85 percent (Leavy and Rheinschmidt, 2010). The Bill and Melinda Gates 
Foundation has awarded WCET a grant to validate the Predictive Analytics 
Reporting (PAR) framework. This project will aggregate data representing 
more than 400,000 student records from six WCET member institutions to 
conduct large-scale analyses of federated data sets within postsecondary 
institutions to better inform loss prevention and identify drivers related to 
student progression and completion. A critical area of focus for this project 
will be identifying factors that affect loss, progression, and completion for 
the 26-and-under demographic in the United States. 

Perhaps as important as developing appropriate instruments to as¬ 
sess programmatic and instructional quality is finding tools that will help 
distance-education leaders to effectively administer, manage, and analyze 
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assessments. EvaluationKit is software that provides a systemic approach 
to these functions that can be used in conjunction with a web portal or 
learning management system, or can stand alone as an assessment system. 
The EvaluationKit system provides "dashboards" that serve as a single 
point of contact for course and program assessment information that 
administrators, instructors, and students can use. The user can customize 
reports, and a user hierarchy makes control of the level and scope of data 
possible. Web hosting and storage are an important part of the package. 
The key to understanding the power of EvaluationKit is that it is merely a 
system or web-based framework for effectively administering assessment 
instruments. Appropriate administrative and academic leaders handle the 
content of assessments as well as the analysis of results from those assess¬ 
ments (Evaluation Kit, 2011). 

The University of Northern Colorado sought to evaluate online courses 
and programs with EvaluationKIT. Most attractive were the quality checks 
and separate reporting mechanisms at multiple levels: student, instructor, 
chair, program director, and dean. The UNC Center for the Enhancement 
of Teaching and Learning (CETL), whose mission is to improve quality in 
the classroom, maintains a strong faculty-centered approach and stresses 
that utilization of the tool and the reports generated are shared only with 
the faculty and, in fact, are faculty generated. The integrity of the center 
depends on its non-punitive approach, and the tool gives summative 
feedback and improvement. At another level, assessment may be modified 
and redirected for the department chair or program director with a design 
aimed at course or program review. Indeed, the dean of the college may 
build another assessment for its purposes, as well. 

LEADING INFLUENCES ON FUTURE QA DECISIONMAKING 

This paper offers a number of perspectives on issues that seem most relevant 
in 2011. Leading QA initiatives, technology and data mining tools, and the 
interrelationships between external and internal accountability, are signifi¬ 
cant parts of the current landscape. There are a number of considerations 
that are likely to figure in the reshaping of that landscape over time, and 
those trends includes globalization of higher education, the increasing role 
of faculty contingency, and the increasing regulation of higher education. 
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Demand for higher education globally is predicted to increase. The 
number of students in higher education worldwide will grow from 120 
million to 150 million by 2025 (Wiley, 2010). There is already an increasing 
recognition of the circulation of academic talent—both students and fac¬ 
ulty—across national borders (Staley & Trinkle, 2011). Some of this takes 
the form of joint degree programs, international branch campuses, and 
distance-education programs, where students stay in their home countries 
and earn a foreign degree. Transnational education (TNE) is part of this 
globalization phenomenon and has created challenges to quality assurance 
efforts, as even many signatories to the Bologna Accord do not have quality 
standards for TNE-related programs. Many internationally recognized ac¬ 
creditation organizations are stepping forward to guide accreditation and 
quality assurance agencies, including UNESCO/OECD (United Nations 
Education, Scientific, and Cultural Organization), INQAAHE (International 
Network for Quality Assurance Agencies in Higher Education), and ENQA 
(European Association for Quality Assurance in Higher Education). As TNE 
programs create increasing numbers of multilateral academic relationships, 
US distance-education leaders will need to gain and maintain awareness 
of new and developing guidelines (Bennett, 2010). 

An issue often left unspoken in higher education discussions is the 
steady increase in faculty contingency. Most distance-education administra¬ 
tors have utilized part-time and non-tenured faculty. Though they do not 
come without costs, these faculty provide instructional staffing options that 
are often more flexible and affordable than a permanent corps of faculty. It 
is worth noting that faculty contingency is not only a distance-education 
phenomenon but also a growing trend in higher education generally. A 
recent article noted that in 1975, approximately 43 percent of all postsecond¬ 
ary faculty were on contingent appointments; in 2010, that number rose to 
approximately 75 of all postsecondary faculty (Maisto & Street, 2011). The 
implications of a shrinking proportion of permanent faculty are significant. 
For one, faculty are likely to play a reduced role in many higher educational 
functions, ranging from instructional decision making to quality assurance. 
This can result in different views on what factors determine quality and 
who assures that quality. 
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The current regulatory environment for distance-education educators 
and administrators is complex and difficult. Regulations that the US De¬ 
partment of Education issued in October 2010 cover topics such as state 
authorization to deliver distance-education programs out of state and 
confirmation of non-degree programs as leading to gainful employment 
(see US Department of Education, 2011). These regulations have come 
about mainly from the concerns regarding the financial and operational 
practices of some for-profit providers, but the regulations have ensnared 
public, non-profit, and for-profit providers alike (WICHE Cooperative for 
Educational Telecommunications, 2011). 

Given the complexity, expense, and staff time required for compliance, 
it is difficult to discern any silver lining in the cloud of regulation. One posi¬ 
tive aspect is that the regulations are causing us to think about foundational 
questions regarding distance education and administration. For example, 
when state authorization regulations cause us to look deeply into who and 
where our students are, we may come out knowing more about our students 
and their needs. Similarly, gainful-employment requirements can make us 
connect the programs we develop to jobs, grounding distance educators 
and administrators more firmly in the work of preparing graduates for an 
increasingly challenging work environment. 

Judging by the content of the last issue of the Continuing Higher 
Education Review, distance education features prominently as a concern of 
continuing education, and its role is becoming so great that eventually it 
may be difficult to discern whether distance education is part of continu¬ 
ing education, or continuing education is part of distance education. No 
matter what its role, distance education must inevitably include quality 
assurance as a key issue. 
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